Evaluating Interactive Dialogue Systems: Extending Component Evaluation to Integrated System Evaluation
نویسندگان
چکیده
This paper discusses the range of ways in which spoken dialogue system components have been evaluated and discusses approaches to evaluation that attempt to integrate component evaluation into an overall view of system performance. We will argue that the PARADISE (PARAdigm for Dialogue System Evaluation) framework has several advantages over other proposals.
منابع مشابه
A New Statistical Model for Evaluation Interactive Question Answering Systems Using Regression
The development of computer systems and extensive use of information technology in the everyday life of people have just made it more and more important for them to make quick access to information that has received great importance. Increasing the volume of information makes it difficult to manage or control. Thus, some instruments need to be provided to use this information. The QA system is ...
متن کاملIdeas for Extending the Approach to Evaluating Health in All Policies in South Australia; Comment on “Developing a Framework for a Program Theory-Based Approach to Evaluating Policy Processes and Outcomes: Health in All Policies in South Australia”
Since 2008, the government of South Australia has been using a Health in All Policies (HiAP) approach to achieve their strategic plan (South Australia Strategic Plan of 2004). In this commentary, we summarize some of the strengths and contributions of the innovative evaluation framework that was developed by an embedded team of academic researchers. To inform how the use of HiAP is evaluated mo...
متن کاملEvaluating Dialogue Strategies in Multimodal Dialogue Systems
Previous research suggests that multimodal dialogue systems providing both speech and pen input, and outputting a combination of spoken language and graphics, are more robust than unimodal systems based on speech or graphics alone (Andr ́e, 2002; Oviatt, 1999). Such systems are complex to build and signifi cant research and evaluation effort must typically be expended to generate well-tuned modu...
متن کاملA Three-Tiered Evaluation Approach for Interactive Spoken Dialogue Systems
We describe a three-tiered approach for evaluation of spoken dialogue systems. The three tiers measure user satisfaction, system support of mission success and component performance. We describe our use of this approach in numerous fielded user studies conducted with the U.S. military.
متن کاملReview of ranked-based and unranked-based metrics for determining the effectiveness of search engines
Purpose: Traditionally, there have many metrics for evaluating the search engine, nevertheless various researchers’ proposed new metrics in recent years. Aware of this new metrics is essential to conduct research on evaluation of the search engine field. So, the purpose of this study was to provide an analysis of important and new metrics for evaluating the search engines. Methodology: This is ...
متن کاملذخیره در منابع من
با ذخیره ی این منبع در منابع من، دسترسی به آن را برای استفاده های بعدی آسان تر کنید
عنوان ژورنال:
دوره شماره
صفحات -
تاریخ انتشار 1997